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Executive Summary 


Machine vision is used in most manufacturing industries to boost quality, productivity, performance and is 
becoming a real differentiator for machine builders in the Industry 4.0 era. This white paper defines the 
process of machine vision and then seeks to answer three key questions that optimize a machine vision 
solution for high performance and high precision machines that demand determinism or hard real time 
computing. 


The first question that must be answered in the decision tree that a machine builder faces explores the 
tradeoff between smart cameras and PC-based machine vision software and highlights why PC-based 
solutions are superior in the Industry 4.0 context. 


Given that PC-based machine vision software approach fits the market of the machine builder, the next 
decision to explore is the tradeoff between commercially available machine vision software and open source 
based OpenCV machine vision. 


The final question to be answered is only relevant to precision machinery but the tradeoff of performing 
the image processing deterministically (sometimes referred to as in hard real time) or not. If image 
processing is not executed deterministically, the machine builder misses a real opportunity to dramatically 
improve performance and quality concurrently. Normally, there is a tradeoff between performance and 
quality, but hard real time or determinism opens a door to improve both which gives the machine builder 
a competitive advantage. 


Purpose and Processes of Machine Vision in High 
Performance Machines 


The Purpose of Machine Vision 


Machine vision is used in most manufacturing industries to boost quality, productivity, performance and 
efficiency. It combines the ability of a camera with the processing power of a computer to make decisions 
about the position, quality, and completeness of a product. 


The role of software in machine vision systems 


Typical machine vision applications in an automation setting range from inspection or defect detection to 
complex 3D metrology or robot guidance. In most implementations though, the fundamental tasks that 
must be supported in software are generally similar represented by the following set of processes: Imaging 
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and Acquisition; Processing and Analysis, Communication and Action. Each process contributes to the next 
and feeds the machine automation with the recommended action and then the process repeats. 


The global machine vision market can be categorized based on type, components, product, application and 
end-users. In terms of type, the global machine vision market can be categorized into 1D machine vision, 
2D machine vision and 3D machine vision. Based on component, the market can be divided into Hardware 
and Software. In terms of product, the global machine vision market can be segmented into Smart camera- 
based and PC-based machine vision systems. The machine vision market can be segmented into inspection, 
gauging, guidance, identification, and others, in terms of application. In terms of end-users, the global 
machine vision market can broadly be categorized into automotive, pharmaceuticals & chemicals, electronic 
& semiconductor, printing & labelling, food & beverage, pulp & paper, postal & logistics, glass & metal 
and others. 


Imaging & Processing 
Acquisition & Analysis 


Communication 
& Action 





Imaging and acquisition 


The imaging device in a machine vision application is tied so intuitively to the machine vision task that the 
term “camera” is sometimes used to refer to an entire machine vision system (as in, "what kind of camera 
are you using for that inspection?”). Fundamentally however, all machine vision systems have at least one 
device — camera/sensor/detector — that is a discrete hardware component for image formation and 
manipulation. Virtually all these devices use dedicated electronics and firmware to manage the low-level 
sensing and image transfer, but for our discussion the important software functions are to access and 
control the parameters that configure how the device forms the image and provide methods to transfer 
and subsequently acquire the image on a computing platform. 


Processing and analysis 


The very core of every machine vision application is the software that performs the actual processing and 


wu 


analysis of the image. At this point specific software tools (“algorithms,” “operators,” etc. depending on the 
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terminology used by the application or library vendor) are configured or programmed to perform specific 
analysis on the pixel-based data in the acquired image. 


Whether programming at a low-level, or configuring an application, this development step is where the 
creative and sometimes difficult machine vision engineering takes place relative to the software component 
of the system. The developer selects and combines the tools necessary to execute the application. By way 
of further explanation, common tools found in most software implementations could be assigned perhaps 
to a few descriptive categories: preprocessing, blob analysis, edge analysis, search, matching, color analysis, 
classification, optical character recognition and verification (OCR/OCV), and bar or 2D code reading. In some 


wu wu 


cases, the tools might be categorized alternately by function: “inspection,” “measurement,” “reading,” etc. 
Overall, the tool selection varies widely by component or library with some libraries offering literally 
hundreds of individual operators. In both configurable and programmable software packages, tools or 


operators must be combined in order to execute a complete machine vision application. 


Communications and Action 


Ultimately the machine vision system must interface with the outside world. The ability to receive production 
information or execution instructions from the broader automation system, and to provide results and 
Statistical data to the process is critical to both the success and value of the machine vision system. A 
configurable machine vision software package often provides various communication and result processing 
tools, and at design time it might be important to consider whether the interfaces available in the software 
are well-suited for a specific automation environment. In programmable systems, the task of 
communication and results processing might be external to the actual machine vision software and handled 
by other software or code. 


The successful application of machine vision technology involves an intricately and carefully balanced mix 
of a variety of elements. While the hardware components that perform the tasks of image formation, 
acquisition, component control, and interfacing are decidedly critical to the solution, machine vision 
software is the engine “under the hood” that supports and drives the imaging, processing, and ultimately 
the results. This discussion will detail the various ways software impacts industrial machine vision systems 
and how it is applied to achieve a complete solution within different component architectures. We also will 
take a brief look at general design and specification criteria and current trends in software that might 
contribute to greater reliability in some machine vision tasks for industrial automation. 


Selecting the Optimal Machine Vision Architecture 


As described above, every machine vision application needs a camera, image-processing software and 
hardware, lighting, and communication interfaces to succeed. There are two principal architectures and 
there are significant trade-offs between. There is a smart-camera approach and the PC-based image 
processing software approach. The final decision on architecture is typically determined by application 
complexity, acquisition and operation costs, support, upgradeability, and reuse. 
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Architecture defined by where the image processing executes. 


Smart Camera Architecture 


A smart camera offers a machine vision architecture and, in addition to image capture circuitry, the smart 
camera is capable of extracting application-specific information from the captured images, along with 
generating event descriptions or making decisions that are used in an intelligent and automated system. 
Most of the work is performed on the camera itself and instructions for actions are passed to a controller. 





Smart Camera Passes Decision To Monitoring PC 


A smart camera is a self-contained, standalone vision system with built-in image sensor in the housing of 
an industrial video camera. It contains all necessary communication interfaces — for example, Ethernet, as 
well as industry-proof 24V I/O lines for connection to a PLC, actuators, relays or pneumatic valves. It is not 
necessarily larger than an industrial or surveillance camera but usually is. 


A capability in machine vision generally means a degree of development such that these capabilities are 


ready for use on individual applications. This architecture has the advantage of a more compact volume 
compared to PC-based vision systems and often achieves lower cost, at the expense of a somewhat simpler 
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user interface. Less powerful versions are often referred to as smart sensors. Because it is self-contained 
and proprietary, there is no opportunity to add new features or 3rd party software like Artificial Intelligence 
to augment the image processing. 


PC-Based Machine Vision Software Architecture 


In contrast to smart cameras, the other commonly implemented machine vision systems have a completely 
open architecture where a general-purpose, less-expensive imaging device (that is, an IP camera or sensor 
appropriate to the needs of the application) is linked to a standard computing platform like a PC running a 
commercial operating system like Microsoft Windows or Linux. The interface from camera to computer 
most often uses one of several image-transfer protocols -- some protocols are proprietary, and some are 
industry standard. In either case, the actual image processing and analysis is performed on the PC rather 
than on the camera. 





IP Camera Passes Image To Smart Controller PC 


Launched in 2006, Gigabit Ethernet or GigE has become the most widely adopted transmission standard for 
image transmission. Gigabit Ethernet offers the widest technological flexibility with regard to bandwidth, 
cable length, and multi-camera functionality. It is the right choice for data rates of up to 120 MB/s and a 
maximum cable length of 100 meters. And it makes complex setups with several cameras very simple — just 
plug in a TCP/IP CAT5 cable. 


GigE has the fastest growing interface for digital cameras in the field of industrial image processing. It is a 
universally applicable digital interface, which can replace analog devices in almost every application. 


In addition to the physical interface defined by the Gigabit Ethernet standard, GigE Vision Standard extends 
and supports easy integration in all image processing programs via the use of software “libraries”. Also, 
because the exchange of GigE Vision compatible cameras can be performed without changing the 
application software, new investments and follow-up costs can be estimated and well-planned with cameras 
based on the GigE Vision Standard. 
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The cost of these Open PC-architecture systems has dropped significantly with the introduction of the GigE 
Vision standard that creates a more consistent way for cameras to connect to PC software. GigE replaces 
the need for a proprietary vision network and IO Card and simply utilizes standard CAT5(TCP/IP) cable and 
the PC NIC card. The strength of the Open PC-architecture is that it can also run 3rd party software, home 
grown applications and Artificial Intelligence applications on that same PC as the images are being 
processed and this can enrich the information delivered by the image processing. 


Smart Camera Advantages 


Simple smart cameras generally cost less to purchase and maintain than their PC-based counterparts, are 
easier to use, and simpler to integrate within an existing automation system. These advantages are due to 
their relatively uncomplicated design. Compared to PC systems, smart cameras have more compact 
dimensions and fewer components. Since they have a smaller number of moving parts and rarely generate 
high temperatures, maintenance costs are kept at a minimum. Likewise, the software programs designed 
for smart cameras are also relatively simple, meaning that advanced computer skills are not always 
necessary for implementing a smart camera system. 


Smart cameras can be designed with “open embedded” processing technology that eliminates the need for 
certain peripheral devices, such as a frame-capture card or an external computer. This streamlined system 
reduces costs while retaining functional capacity and, if coupled with an advanced processor, can rival some 
PC systems for power. In addition, the open embedded format is free of interfaces, which are usually needed 
to link multiple components into an operational network. Open embedded smart cameras are considered 
standalone vision systems, as they can generally handle tasks with minimal reliance on secondary devices. 


Many cameras are equipped with communication hardware (Ethernet, RS232, etc.) that can transmit data to 
mainframes or other storage devices for future use. If the camera has limited processing ability, transferring 
information can augment processing speed by shifting administrative duties to an external device. 


Smart Camera Disadvantages 


Due to their simple, compact design, smart cameras usually have less processing capability than PC-based 
systems. This limits the range and number of tasks that a smart camera can handle. Since they have difficulty 
managing more sophisticated algorithms, complex images or operations requiring rapid analysis are 
typically outside their scope. 


Smart cameras often use proprietary hardware, making replacement or modification of parts a challenging 
process. When coupled with a compressed design, this feature severely limits the degree to which smart 
cameras can be upgraded. For similar reasons, expanding a smart camera's functional range to handle 
different or additional tasks can be problematic. 
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To cram in more functionality, some smart cameras have become quite expensive and still do not provide 
the openness of the PC architecture. 


PC-based System Advantages 


As previously mentioned, PC-based vision systems generally have greater processing power and are capable 
of handling complex operations at relatively high speed. This broader range of capabilities also enables PC 
systems to compensate for unexpected variables in certain tasks. For example, products on an assembly 
line tend to accumulate slight variations over a period of time, sometimes measured in months. Because 
the deviations occur gradually, they may not be detectable by a system with limited processing power, but 
a sophisticated computer vision network can perceive and mark the change. 


Unlike smart cameras, most PCs are upgradeable and can have components swapped with relative ease. 
This versatility makes a PC system highly customizable, as it can have newer or more application-specific 
hardware installed to specialize on a certain task or have its general range of functions expanded. For 
instance, a PC-based vision system could initially be employed for identifying and measuring components 
on an assembly line. Its duties could then be extended through installation of a software and hardware 
package to control a robotic arm that would remove any flawed products from the line. 


Also, the Smart Cameras can become quite expensive. Since all the image processing and direct machine 
control is performed on the PC, the IP camera used to collect the image can be far less expensive. 


Finally, the ultimate objective of machine vision is to share the execution guidance with the machine 
controller. If the machine vision and the machine controller are on the same PC then the integration effort 
is minimal. 


PC-based System Disadvantages 


The strength of the PC-based system also contributes to its weakness. Since the PC-based system is 
upgradeable, flexible, and capable of executing additional applications besides image processing, these 
systems can become complex and must be managed. This is stark contrast to a smart-camera which is 
locked down, not upgradeable, inflexible, not directly able to be integrated with other applications so does 
not require much management. 


Industry 4.0 Mandates The PC-Based Machine Vision Architecture over Smart 
Camera 


Throughout the manufacturing and industrial sectors, machine vision is having a significant impact. Machine 
vision is no longer limited to brick-and-mortar facilities but is finding its way into environments like farms, 
airports, and roadways. Burgeoning Industry 4.0 systems rely on machine vision-enabled devices to report 
what they capture to the cloud so artificial intelligence can make informed decisions. 
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There is no “one size fits all” solution when it comes to machine vision hardware, which complicates 
hardware and software specification. However, while Smart Cameras will continue to be an important 
solution, the Open, PC-based architecture is far better suited to support the growing demands of the loT 
and Industry 4.0 era and beyond. Systems that can enrich or more tightly integrate the image processing 
results with other data stand to develop a competitive advantage. 





Image data contributes to loT on Smart Controller PC 


In other words, to optimize the smart edge device, Industry 4.0 requires multiple controllers and 3rd party 
software to run concurrently on an industrial PC because this facilities integration for information sharing 
and on-the-fly information consuming to optimize the machine, the work cell and even the factory. Smart 
Cameras only support vision but cannot run the Al or other controller applications that are required for a 
vision directed motion application like autonomous robots or other coordinated demands of the loT era. 
As a result, the Open PC-based Machine Vision Software applications are the superior architecture to the 
smart-camera architecture for sophisticated and loT applications because co-operative third-party software 
and communication infrastructure are readily available on the PC. As a result, machine builders are now 
adopting the open PC architecture more rapidly in order to insure a path to the future. 


Selecting PC-Based Image Processing Software 


Machine Vision Infrastructure and Image Library Solutions 


Before jumping into a discussion about the features to consider when selecting the image library that is the 
basis for analysis and processing in a machine vision software solution, it is good to understand the three 
fundamental alternatives available to a machine builder. The machine builder can purchase a commercially 
available but proprietary third-party image libraries and image processing infrastructure solutions, can rely 
on an open-source library known as OpenCV combined with GigE, or can develop their own image library 
solution. 
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First, there are several 3rd party Image Library and image processing packages that are commercially 
available for resale. The solution can be loaded on a PC and integrated with a machine controller. These 
products have time tested algorithms for image processing (e.g. edge detection) and thousands of 
deployments. 


Second, OpenCV is the open-source equivalent of these commercially available. As is the case with many 
open-source solutions, the machine builder has to take more responsibility for integrating the image 
processing and analysis libraries into the application, but there is no run time charge like the commercial 
alternatives, which can be very expensive. 


The third alternative is to just develop image processing and analysis without the aid of image libraries 
provided by the market. While it is challenging and seems daunting, some machine builders feel that the 
image processing can become a competitive advantage not only because there is no runtime charge like a 
commercial solution, but they can optimize the image analysis for their specific machine. 


Machine Vision Software for Image Processing on an PC in the Market 


Until recently, the choice of image processing and analysis software was limited with many companies 
merely offering callable libraries that performed relatively simple image processing operations. These 
included point processing operations such as image subtraction, neighborhood operations such as image 
filtering and global operations such as Fourier analysis. 


While useful, developers were tasked with understanding each of these functions and how they could 
provide a solution to a machine vision task such as part measurement. Often, the need to build software 
frameworks to support such libraries made developing such programs laborious and time consuming. In 
terms of practical implementation, one construct for machine vision software can be described as an 
application that “configures” the system components and how they execute machine vision functions and 
tasks. These apps tend to have graphical user interfaces (GUI) devoted to “ease of use” with intuitive and 
graphically manipulated application configuration steps. 


As for the infrastructure, as noted above, many commercial image processing software solutions offer their 
own proprietary frame-grabbing and IO infrastructure but thanks to the power of standards GigE Vision has 
become the default transmission technology standard. 


In other words, most commercial and open-source image processing software offers two components: 1. 
The infrastructure for acquiring the image and for sharing the result of any image analysis, 2. the libraries 
for performing the actual image analysis. 


As a result, Machine Vision software is designed for open architecture machine vision systems that is fully 


programmable. Typically targeting users with suitable levels of experience in computer programming in 
languages like C, C++, C#, .NET and others, these software products might be called “software development 
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kits” (SDK) or “libraries,” and contain an extensive selection of low- and medium-level operators 
(“algorithms”) that when properly combined perform tasks ranging from very basic to extremely complex. 
In many cases also, libraries designed specifically for machine vision application development feature an 
integrated development environment (IDE) or even a configurable software application built on the 
underlying tools found in the library. These product extensions can make the development process much 
easier in applications that use these libraries without sacrificing the full functionality of the individual tool 
or algorithm, and in some cases offering a path to migrate the configured application code or script to a 
lower-level programming language automatically. 


Summary of Commercial Versus OpenCV Image Analysis Software 


The decision criteria for selecting commercial or OpenCV image analysis software is actually a simplified 
version of any open-source discussion. There are benefits and risks associated with both a commercial 
package and with an open-source packages but they emerge in different ways. 


With the commercial package, much of the integration work is completed, the libraries are robust, the user 
experience is simplified, and the support and training are available to help users get to market successfully. 
The price that a machine builder pays is that each machine vision deployment is rather expensive. By 
contrast, with OpenCV, the benefit is that there is little or no royalty cost for deploying an open-source 
solution, but there is integration work required and there is little support available. The question is whether 
the machine builder wants to pay an upfront engineering cost or pay a steep royalty fee and be locked into 
a vendor because the image libraries are proprietary. 

To find a list of specific commercially viable image processing software packages, please see the article 


entitled Image processing software: Software packages offer developers numerous options for 
machine vision design by Andy Wilson on www.Vision-Systems.com website. 


Value of Adding Real-Time, Deterministic Machine 
Vision 


Deterministic capability delivered by a Real Time Operating System (RTOS) adds game-changing 
functionality to vision systems by improving performance, quality and throughput. 

In a deterministic system, the image capture starts immediately after a trigger signal has been sent to the 
camera. Image data must be acquired, transmitted, and evaluated within a deterministic, pre-set timeframe. 
The requirements for the maximum acceptable reaction time between trigger signal and image acquisition 
can vary from microseconds up to seconds. For more and more industrial cameras, hard real-time images 
are a prerequisite. 
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Real Time Vision on SMT Machine Improves Quality 


When inspecting components in a production process, for example, the components are transported on 
the conveyor belts at high speed. For a precise inspection, the camera must acquire the images as quickly 
as the components are being transported. Timing this precisely requires low latency: a small-time delay 
between receiving the trigger signal and acquiring the image. Further, the time delay must not vary; no jitter 
can affect the moments of image acquisition. For an application with high image rates (e.g. 300 images per 
second), the required latency times can only be microseconds. And this task demands the use of a RTOS. 


This is especially true for vision directed motion applications like robotics. For example, vision can assist in 
the orientation of a chip in the surface mounter chip insertion machine or can guide a robot to weld a seam 
on a 4-door car and then, on the fly, can change over to weld a seam on a 2-door car. If hard real time 
vision was not part of the deterministic motion loop, then the latency would be so great that the 
performance would have to be slowed dramatically to ensure quality. 


And there are many more examples in the world of robotics, where machine vision systems have long been 
used to “teach robots to see.” After all, robot arms can only perform their high-precision gripping and 
positioning maneuvers once cameras and image processing tell them precisely where to move. Small, light 
industrial IP cameras that just plug into a TCP/IP CAT 5 cable are typically used in robots. The devices are 
ideal for “pick and place” applications: gripping, mounting, and positioning tasks during computer chip 
assembly. Industrial cameras are also needed for measurement and quality controls of the various product 
characteristics between and during all production steps. 


Safety during the production process is always a prime concern for robotics applications. Previously, barriers 
were built to prevent workers from entering the danger zones containing the rotating robots. The structures 
were expensive, cumbersome, and inflexible. IP cameras have changed the situation fundamentally. The 
network technologies can now be placed around the robots to create a “virtual cage” based on markings 
on the floor. The camera monitors those marked zones and is authorized to stop the machine if material, 
or an employee, enters the robot's working area. These applications only execute dependably 100% of the 
time if they run on a RTOS, so that RTOS feature offers competitive separation for Machine builders that 
embrace it. 
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Competitive Landscape for Real-Time Image Processing Software 


In surveying the market, machine builders will discover that there are three PC-based machine vision 
software solutions available for adding a deterministic Image processing capability to a next generation 
machine. Two of these solutions are based on world-class, commercially proven, non-deterministic image 
processing software. The image processing vendors do not include any real-time on their own but instead 
refer interested parties to the RTOS vendor who supplies a third-party plug-in to add the real-time. 


The third option for finding adding real-time image processing to a machine is to use commercially available 
solutions that port the open source OpenCV machine vision to a real time operating system (RTOS) like 
IntervalZero’s RTX64 RTOS to deliver on the promise of Real Time Machine Vision. Real Time Machine 
Vision solutions can be downloaded from the IntervalZero RTX64 Vision page or also found at the 
KINGSTAR Vision page. 


As is the case with open source, there is a price to pay in the form of integrating new machine libraries, but 
the benefit is that there is no cost for the Image Libraries. By contrast, the commercial vendors charge a 
handsome fee for the image libraries and once a machine builder starts using the image libraries they 
become locked into that vendor for years to come. Just to net out the competitive landscape, here is a 
summary of the advantages and weaknesses of each approach. 


KINGSTAR Vision or Commercial 
RTX64 Vision Image Processing 
Software 


GigE Vision 
Infrastructure 





Deterministic Image Require 
Processing (RTOS) 3°¢ Party 
Open Standard Image Vendor 

Library e in 

Out-of-the-Box 

Experience 
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Summary 


Machine vision is used in most manufacturing industries to boost quality, productivity, performance and is 
becoming a real differentiator for machine builders in the Industry 4.0 era. Having an open-source, 
deterministic machine vision solution on PC offers strategic value to a machine builder that is seeking to 
differentiate its next generation machine. 

There are two prevailing architectures that can deliver machine vision — the smart camera and PC-Based 
solutions. While the smart camera can deliver amazing functionality, they tend to be expensive islands of 
automation, so they are not as versatile as a PC-based architecture. This is especially true in the Industry 
4.0 era which will demand more edge analytics and data sharing with the cloud. The ultimate objective of 
machine vision is to share the execution guidance with the machine controller. If the machine vision and 
the machine controller are on the same PC then the integration effort is minimal. The Smart Camera is on 
a different hardware than the machine controller so that does reduce the ability to have a deterministic 
system that is integrated with the machine controller itself. 


Further, machine builders are discovering that deterministic or hard real time image processing is a feature 
that can deliver competitive separation because set up times are reduced, while quality and performance 
are dramatically increased versus a non-deterministic approach. 


Finally, while there are two solid commercially available image processing solutions that are targeted for 
the PC, they are not deterministic and they required a third party plug-in from an RTOS vendor. The cost 
starts to build as more components are added. Alternatively, there is an open-source solution for machine 
vision, OpenCV. OpenCV now runs on a world class RTOS, IntervalZero’s RTX64. Since OpenCV is open 
source, there is not charge for the image processing libraries. While there is work involved with customizing 
the libraries at the outset and costs for the PC and Operating Systems, this real-time machine vision solution 
is the most open and lowest total-cost-of-ownership solution for a machine builder. The open system 
approach allows the machine builder to deploy both the real-time machine vision with OpenCV and the 
machine controller on a single PC and thus save money from controller consolidation. 
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